release: 0.3.0: production hardening and checks that prove the data paths - #21

…tunnel and access

- exemplar destination trace_id -> traceID, matching what Tempo emits
- HighErrorRate filters span_kind=SPAN_KIND_SERVER on both sides; internal
  ASGI child spans inflated the denominator ~3x
- demo load hits /work twice so the demonstrated error rate (~10%) actually
  crosses the 5% threshold instead of sitting on it
- infrastructure-logs dashboard queries service_name (the label Loki actually
  indexes); drop the env variable no templating query can populate
- storage-s3 overlay re-declares commands with -config.expand-env=true, or the
  documented activation reads the \${S3_*} placeholders literally
- drop tempo datasource search.hide, restates the default

- file-backed exporter queues on a new otel_queue volume; buffered telemetry
  survives collector restarts, retry/queue settings pinned to the documented
  five-minute window
- mem_limit on every service, sized from observed usage; collector
  memory_limiter switched to absolute values tied to its container limit
- prometheus healthcheck with service_healthy ordering for collector and
  grafana (loki/tempo are distroless, no probe binary)
- scrape grafana, loki, tempo, and alertmanager so TargetDown covers them
- loki indexes only service.name; service.instance.id becomes structured
  metadata instead of minting a stream per sender restart
- tempo metrics-generator capped at 50k active series
- grafana 13.1.4 (CVE-2026-13438, CVE-2026-17183), loki 3.7.6, tempo 3.0.3,
  prometheus v3.13.2, node-exporter v1.12.1, cloudflared 2026.8.2
- secure cookies behind GRAFANA_COOKIE_SECURE (up-tunnel enforces it);
  opt-in per-user Grafana login via Cloudflare Access JWT
- no-new-privileges on all services; lint containers run --network none;
  infra-validate keeps tfstate/tfvars out of the container
- up-tunnel also guards GRAFANA_ROOT_URL and warns on empty HEARTBEAT_URL;
  logs/ps/tail see demo services; loki/tempo config checks join just check;
  smoke asserts every dashboard provisioned

- run just infra-validate; tofu fmt alone let invalid resources merge green
- build the demo image so a broken demo app fails CI
- pin actions/checkout and setup-just to commit SHAs

…config

- delete relab-api.json; the \$service pickers on Service Health and Logs
  Overview already cover it and scale to future services
- delete demo/__init__.py, never copied into the image or imported
- allowUiUpdates: false, the :ro dashboards mount made UI saves fail anyway;
  drop provisioning keys that restate defaults

- fastapi 0.141.1, uvicorn 0.52.4, otel 1.44.0/0.65b0, uv 0.12.1
- python:3.14.7-slim in both stages for reproducible builds
- run as nobody

- README: prerequisites, accurate volume list and just-check coverage,
  dashboard JSON editing workflow, up-tunnel guard description
- RUNBOOK: durable queue and just-up volume init, memory ceilings, JWT
  access setup with the aud-pinning caveat, manually-tracked tool images
- ONBOARDING: loki's new label surface (service_name only)
- CHANGELOG: Unreleased entry covering the six weeks since 0.2.0 and this
  batch

- add ADR 0002: hub-and-spoke observability for CML projects, superseding
  ADR 0001's span-metrics RED decision
- move the August 2026 review handover in from the Relab repo as
  docs/HANDOVER.md, with an owner/review-date tripwire
- point the README at both

Verified against the running stack with Relab reporting.

- key the rule on project/env as ordinary series labels: Alloy attaches the
  identity labels to the series, not as OTLP resource attributes, so target_info
  carries none of them and the previous expression could never match
- correct the environment to staging; no production series exist
- record the live findings in the handover: cAdvisor sends only the root cgroup,
  so ContainerRestarting matches nothing and the crash-loop incident is still
  undetected; 30% of OTLP metric writes were being rejected as out-of-order
  until the window this branch carries was actually loaded

- remove the generator, its remote-write path and the cardinality backstop
- drop Prometheus' remote-write receiver, which existed only for it
- unwire tracesToMetrics/serviceMap/nodeGraph rather than leave an empty
  service graph reading as "no traffic"

- provision rules, contact points and the routing tree from
  config/grafana/alerting/; one engine that can query Loki as well
- drop Alertmanager, its url_file entrypoint hack and its volume
- add AlertDeliveryFailing, and make ALERT_WEBHOOK_URL a hard guard in
  up-tunnel: empty delivery fails silently while the heartbeat stays green

Template variables are stream selectors and label_values() cannot see
structured metadata. All four are bounded, so indexing them costs nothing.

- replace infrastructure-logs and logs-overview with one logs dashboard
- rework service-health and stack-health onto the OTLP-native series

- one Alloy agent per host: container logs, host metrics, cAdvisor, GPU
- bootstrap renders the keystone rule, regenerates the coverage backstop,
  creates healthchecks.io checks and prints the .env and vendoring curls
- replace the ONBOARDING templates that pushed straight to Loki, which
  needed Loki exposed without authentication

- mark ADR 0002 complete and release the spoke-watchdog tripwire
- delete HANDOVER.md: it described a transition, not a system
- point onboarding at templates/README.md

…ed rule count

- config/alerts/ went with Alertmanager, but Prometheus and the promtool
  validator still bind-mounted it, so Docker kept recreating it root-owned
  and promtool checked a config with no rules
- assert in `just smoke` that as many alert rules provisioned as there are
  on disk; a malformed file provisions none of its own group, so a non-zero
  check would miss it

A project bootstrapped for one environment read as covered while its other
environments had no keystone rule — the same silent gap the rule exists to
close, one level in.

- read the pair back from a COVERS marker in each rendered file rather than
  splitting the filename, which is ambiguous when either half has a dash
- match with `unless on (project, env)` over the rendered pairs
- correct the template header: rendered flat, and the file is coverage.yaml

- nothing detects an empty ALERT_WEBHOOK_URL: AlertDeliveryFailing routes to
  that same webhook and the heartbeat stays green, so the up-tunnel guard is
  the only control
- ONBOARDING no longer carries the Loki-driver and Alloy templates
- move the Cloudflare seat-count caveat to the access bullet it qualifies

- overlays are host config: every recipe (up, down, logs, ps, backup) acts on
  the same COMPOSE_FILE set, so tunnel services are visible to logs/ps on the
  hosts that run them
- drop 'just up-tunnel': 'just up' runs the exposure guards automatically
  whenever the tunnel overlay is active
- document the mechanism in .env.example, the README, and the runbook"

- render the ProjectsUncovered rule from templates/alerting/coverage.yaml.tmpl
  instead of a shell heredoc, and re-render every covered pair on each
  bootstrap run so template fixes propagate to all projects
- collapse the identical FIRING threshold node (rules.yaml) and the exporter
  queue/retry blocks (otel-collector.yaml) into YAML anchors
- share the compose file set and dashboard paths via justfile variables, run
  'docker compose config --images' once in check, and verify dashboards in
  smoke with one search call instead of per-uid polling loops
- scope the Service Health logs panel to the selected service and use the
  indexed label_values(project) in the logs dashboard
- x-logging anchor in the telemetry overlay, ping_fail() helper in
  run_scheduled.sh, HC_JOBS override for the healthchecks job list
- fix docs contradicting the config: Loki's four identity index labels,
  Tempo's removed metrics generator, the Alertmanager removal, and the now
  centrally provisioned GPU/container dashboards"

- drop the JWK-set fallback domain: GRAFANA_JWT_AUTH now requires
  CF_ACCESS_TEAM_DOMAIN and CF_ACCESS_AUD, with the aud pinned via
  GF_AUTH_JWT_EXPECT_CLAIMS and exported from infra as grafana_access_aud
- pass the tunnel token to cloudflared via TUNNEL_TOKEN env, not argv
- cap_drop ALL on every service; node-exporter read-only with a pids limit
- digest-pin all hub images, matching the client templates
- bootstrap: healthchecks API key via curl config-from-stdin, vendoring
  refuses a tag without templates/ and prints sha256sum -c lines
- grafana healthcheck so compose --wait has something to block on

- aggregating by job alone let a healthy prod service dilute a broken
  staging one sharing the job name below the 5% threshold
- HostDiskSpaceLow names the host via host_name or instance
- wrap long exprs and descriptions as folded scalars

- mirrors the hub collector: a long hub outage now sheds load instead of
  OOM-killing the agent together with its drop counters
- shorten the OTLP endpoint guard message

…use readable secret files

- smoke posts through curl's stdin config; busybox wget put the token in argv
- _expose-guards fails when .env or the tofu state is group/other readable
- the prek gitleaks hook mounts the repo read-only like just lint

TargetDown, HostDiskSpaceLow and HostDiskFilling returned 0 or a negative
value on the state they watch; the shared threshold node fires on > 0.

…E_ALERTS

- opt-in locally, on in CI: costs the rule's for plus one evaluation
- poll() takes POLL_MAX for the longer wait

- set -o pipefail: tar's status was lost behind gzip and a partial archive passed
- restore-check exports COMPOSE_FILE so the host's overlays stay out

The readback loop folded a 401 into 'rule not provisioned'. Fail fast on an
empty password, die on 401/403, escape the password for curl's config.

…es at 100

lookup_dns_record died inside a command substitution, so an API error still
read as 'no record'.

…ction

- Fold the repeated hardening, logging and healthcheck blocks into
  `x-hardened` and `x-healthcheck` anchors; services override after the merge.
- Give every alert rule the same `rule_defaults` anchor, and move the capacity
  group off a 1m interval: nothing there fires in under 15m.
- Stamp `department` as a literal in the collector config. One value has ever
  been correct, and as an indexed Loki stream label a second one would be an
  index dimension, not a filter; the `DEPARTMENT` variable is gone from `.env`.
- Pull the spoke's dummy environment into `spoke_env`, read the Alloy image
  from the template that pins it, and only pull lint images that are missing.
- `bootstrap.sh` reads single keys out of `.env` through one helper and prints
  the vendoring curls from the same list it hashes.
- CI copies `.env.example` with mode 600, or the exposure guards reject it.

- Drop histogram buckets from the Grafana, Loki and Tempo self-scrapes. Of
  those three jobs' 8,045 series, one metric is read anywhere in this repo;
  the buckets were 5,264 of them. `_sum` and `_count` stay, so latency is
  still there to explore.
- Run node-exporter with the same nine-collector allowlist the spokes' Alloy
  uses. The defaults added ~1,000 series of cpufreq, thermal and cooling data
  that no dashboard, alert or runbook reads.

Measured on the hub with one spoke: 14.1k active series to 7.0k, 438
samples/s to 206.

- Keep the four cAdvisor metrics two dashboard panels and the two
  container-lifecycle alerts use. cAdvisor emits ~59 metric families per
  container, and per-container-per-host is the fastest-growing block in the
  series budget.
- Drop Alloy's own histogram buckets: 345 of that job's 760 series, none of
  them read centrally.
- Poll Docker every 15s, so a short-lived one-shot is discovered without
  hitting the socket proxy at the old rate.

The keystone alert and the coverage backstop asked "is this project sending
anything?" by selecting every project-labelled series in the TSDB: a scan that
grows with the fleet, and one that a logs-only or traces-only project is
invisible to, because it reaches no Prometheus series at all.

- Count what arrives at the collector, keyed on the project and env resource
  attributes, and emit telemetry_{datapoints,logs,spans,metrics}_total.
- Point ProjectTelemetrySilent and ProjectsUncovered at those counters.
- Convert the connector's delta sums to cumulative: the Prometheus OTLP
  receiver refuses delta ("invalid temporality and type combination"), which
  would have failed silently at exactly the layer meant to catch silence.
- Count a sender that set no project as "unknown", so an unattributed sender
  raises ProjectsUncovered instead of vanishing.
- Assert the counters in `just smoke`, since two alerts now key on the name.
- Show ingest per project on Stack Health.

Both onboarding templates and the demo set env and never project, so an app
onboarded by following them shipped telemetry no alert or dashboard could
attribute. With the gateway's ingest counters such a sender is now counted as
"unknown" and raises ProjectsUncovered, but the docs are where it should not
happen in the first place.

- Changelog entries for the counters, the halved series count and the
  onboarding fix, and drop the DEPARTMENT upgrade step it no longer has.
- Re-base the cardinality prose on the measured ~7k baseline: the 100k
  ceiling is ~14x it now, not ~7x.
- Verify a spoke with the ingest counter, which sees every signal.

The printed check still selected every series a project sends, which is both
the scan the alert rules moved off and blind to a project that ships only logs
or traces. templates/README.md already says the counter.

CF_ACCESS_TEAM_DOMAIN was the one Access value copied by hand from the
dashboard, and Grafana builds its JWK set URL from it: unset, it fetches
signing keys from a subdomain anyone could claim, which is why the exposure
guards refuse the pair. The team name is account-wide and predates this
config, so read it through a data source and trim the suffix Grafana appends.

The old comment argued a second department could not be a stream label, which
is what the label already is. What the literal actually buys is that it cannot
be misconfigured per deploy, and the trigger for a variable is a second hub,
not federation: a second department federating into this hub means the upsert
has to go, since it would stamp our department onto their data.

A host without the NVIDIA container runtime cannot include the GPU overlay at
all, so tooling that assembles its `-f` list from the host's .env needs a
switch, and every spoke inventing its own name makes one runbook per spoke.
None of the vendored files read it; say so where the name is given.

…lector

- validate reuses _image instead of an inline copy of the jq lookup
- telemetry.spanevents counts one series, nothing reads it per project
- rules, bootstrap and smoke match telemetry_.+_total in one form

- move the queue chown into a one-shot otel-queue-init service, so a plain
  `docker compose up` works and the volume is created labelled. Pre-creating
  it with `docker volume create` left it unlabelled, and `down --volumes`
  then never reclaimed it
- refuse to restore when a stateful volume is missing: `docker run -v` would
  create it empty, making the pre-restore snapshot a tarball of nothing
- drop the dead Alertmanager cleanup step and the stale "use just up" warning
  from the runbook, and note that the alpine pin now lives in two places